Finding Literary Themes With Relevance Feedback

A common task in text analysis is find conceptually-linked passages of text such as examples of themes, imagery, and descriptions of events. In the past, researchers looking to find such passages have had to rely on searching for sets of keywords. However, themes, descriptions, and imagery may surface with many different phrasings, making retrieval based on keyword search difficult. We investigated the application of relevance feedback to this problem. First, we implemented a relevance feedback system for sentence-length text. Then, we evaluated the system’s ability to support gathering examples of themes in the works of Shakespeare. Participants with at least undergraduate backgrounds in English language or literature used either our system (N = 11) or keyword search (N = 12) to retrieve examples of a theme chosen by a professional Shakespeare scholar. Their examples were judged on relevance by our expert collaborator. Our results suggest that relevance feedback is effective. On average, participants with relevance feedback gathered more sentences, and more relevant sentences, with fewer searches than participants with keyword search. However, a larger study is needed to establish statistical significance.